feat(reindex): metadata-only mode to backfill MongoDB missing_params (#2008) by mauripunzueta · Pull Request #2213 · HeliosSoftware/hfs

mauripunzueta · 2026-10-21T18:13:44Z

Closes #2008.

What

A metadata-only $reindex mode that fills in missing_params on MongoDB resources written before #1984, without rebuilding the value index.

{"resourceType": "Parameters", "parameter": [
  {"name": "metadataOnly", "valueBoolean": true},
  {"name": "skipExistingMetadata", "valueBoolean": true}
]}

metadataOnly: same walk, same extraction (search_index_documents_checked) and same version guard (resource_version_filter, upsert: false) as a rebuild. Only the resources.missing_params update runs; nothing is deleted from or inserted into search_index / search_index_contained.
skipExistingMetadata: adds missing_params: {$exists: false} to each update's filter, so resources that already carry the field are left alone.
$reindex-status adds metadataOnly: true and metadataUpdated, the number of resources the updates matched. entriesCreated stays 0. Per-resource failures are reported as they are today.
Validation: 400 when combined with clearExisting, clearOnly or a bulk index rebuild, and 400 for skipExistingMetadata without metadataOnly. It does combine with POST /{type}/$reindex, with idStart/idEnd, and with named resources.
501 on every backend except MongoDB, including MongoDB with search offloaded (mongodb-elasticsearch never reads missing_params). This is checked before a job id is issued.

How

ReindexTarget gains supports_search_metadata() (default false) and write_search_metadata_page() (default: refuse). In metadata mode the driver writes only through writers that support it.
MongoDB: a new write_metadata_page in reindex_pipeline.rs. It reuses extract_one (on the rayon pool for large pages), update_missing_metadata (now with a counted variant) and page_outcomes.
Docs: in docs/mongodb/search-indexes.md, upgrade step 3 now uses metadataOnly, and there is a new "Metadata-only reindex" section. The definition-maintenance procedure still says full rebuild, because its value-index rows were extracted under the old definitions too. The issue suggested this mode for that case as well, but that would leave stale value rows behind.

Acceptance criteria

After a metadata-only run on a pre-feat(mongodb): index missing search parameters on resources #1984 dataset, every live resource has missing_params, and search_index is untouched. mongodb_metadata_only_reindex_backfills_without_touching_the_index unsets the field across a tenant, runs the mode, and asserts that the stored arrays equal what the original writes stored. It also asserts that every search_index + search_index_contained row is byte-for-byte identical, including _id, so a rewrite would show, and that a second tenant is untouched.
:missing=true results are identical before and after. The same test checks birthdate:missing=true and =false both ways.
Documented in docs/mongodb/search-indexes.md.

Also covered: skipExistingMetadata keeps a sentinel array and respects type scope, offloaded MongoDB returns Unsupported, plus REST parse and validation unit tests.

Verified locally

cargo test -p helios-persistence --features mongodb --test mongodb_tests reindex: 81 passed, including the 3 new tests, against a real MongoDB testcontainer.
cargo test -p helios-persistence --features mongodb --lib reindex: 167 passed.
cargo test -p helios-rest --lib reindex: 20 passed.
clippy: no new warnings on the changed lines. The local toolchain flags collapsible_if on existing code.

Not done / follow-ups

skipExistingMetadata skips only the write. Every resource in scope is still read and extracted, because the reindex source has no "field absent" filter. Pushing missing_params: {$exists: false} into the Mongo walk (id phase and catch-up rounds) would make a mostly-backfilled tenant cheap to re-run. I left it out to keep the walk untouched.
Not run at scale. No measurement yet of the metadata-only throughput against a full rebuild on the Synthea corpus.

…2008) A `metadataOnly` $reindex recomputes each resource's `missing_params` with the rebuild's extraction and version guard and writes nothing to search_index or search_index_contained. `skipExistingMetadata` leaves resources that already carry the field alone. - ReindexTarget gains supports_search_metadata/write_search_metadata_page; only MongoDB (search not offloaded) implements them, everything else is 501 before a job starts. - $reindex-status reports metadataOnly and metadataUpdated. - Rejected with clearExisting, clearOnly or a bulk index rebuild; combines with type scope and idStart/idEnd. - Upgrade step 3 in docs/mongodb/search-indexes.md now uses it.

codecov · 2026-10-21T21:05:39Z

Codecov Report

❌ Patch coverage is 98.10427% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/persistence/src/search/reindex.rs	97.75%	6 Missing ⚠️
...rsistence/src/backends/mongodb/reindex_pipeline.rs	98.94%	1 Missing ⚠️
crates/persistence/src/backends/mongodb/storage.rs	83.33%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…#2008) - Driver unit tests: metadata goes only through writers that keep it, failures are recorded, metadataUpdated/metadataOnly in status, the default trait methods refuse, and the validation branches. - MongoDB: a 40-resource page on a multi-thread runtime takes the prepare-pool extraction path; PageFailure::fan_out_metadata replaces the inline closure and has its own unit test.

Conflict: docs/mongodb/search-indexes.md. Kept main's #1999 text, which replaces the manual definition-maintenance procedure this branch edited. #1999 also added a missing_fp argument to with_missing_metadata_update, so the metadata-only update now stamps the fingerprint as a rebuild does.

#2008) Since #1999 a `missing_params` array is trusted only with the current `missing_params_fp`: arrays written by #1984 (v0.2.5) carry no stamp, and every `:missing=true` on their type takes the value-index route. The skip guard was `missing_params: {$exists: false}`, so on an upgraded tenant it skipped every resource (`metadataUpdated=0`) and left the slow route in place. At 100k Observations, `subject:missing=true` stayed at ~970 ms. The guard now skips a resource only when its array exists and carries the fingerprint this extraction stamps. A failed extraction keeps the old guard. Docs: the #1999 upgrade note points at the metadata-only reindex.

smunini · 2026-10-24T17:52:26Z

Review: tested with and without, plus perf

At the PR head (old base `3281155`)

The claims hold: 82 MongoDB reindex integration tests, 171 lib tests and 20 REST tests pass on a MongoDB 7 replica set.

After merging main: two problems, both fixed and pushed

The branch had gone stale against #2181 (the missing_params_fp fingerprint, merged today).

Merge conflict and compile break. docs/mongodb/search-indexes.md conflicted. I kept main's mongodb: detect SearchParameter definition changes and invalidate stale missing_params automatically #1999 text, which replaces the manual definition-maintenance procedure this branch edited. with_missing_metadata_update now takes missing_fp, so metadata_only_update passes docs.missing_fp and stamps the fingerprint as a rebuild does (6f0c95e, and a second main merge in 42225e3).
skipExistingMetadata skipped the data that most needs the backfill (cd8a2a8). Since mongodb: detect SearchParameter definition changes and invalidate stale missing_params automatically #1999, an array is trusted only when it carries the current stamp. Arrays written by feat(mongodb): index missing search parameters on resources #1984 (v0.2.5) have none, so every :missing=true on their type takes the value-index route. The guard was missing_params: {$exists: false}, so on an upgraded tenant it skipped every resource. The guard now skips only when the array exists and its missing_params_fp equals the stamp this extraction produces. A failed extraction keeps the old guard. The new test mongodb_metadata_only_skip_existing_restamps_unstamped_and_stale_arrays covers an unstamped array, a stale stamp and a current stamp, which must be skipped. It fails without the fix (metadataUpdated 0, expected 2). The docs now say what is skipped, and the mongodb: detect SearchParameter definition changes and invalidate stale missing_params automatically #1999 upgrade note points at metadataOnly instead of a full $reindex.

After the fixes: 111 MongoDB reindex/missing integration tests, 171 lib tests and 20 REST tests pass. Clippy is clean with the CI flags. Running the lib tests in parallel, a different log-capture test fails each time (a_failed_page_closes_…, a_run_logs_…, a_cancelled_run_…). All 171 pass with --test-threads=1, so this looks like the shared tracing-capture race, not this change.

Performance: 100k Observations (1.8M `search_index` rows), MongoDB 7

Run	Time	`metadataUpdated`	Index rows
Full `$reindex` (main's only remedy)	106–146 s	–	rewritten, 1.8M
`metadataOnly` on an upgraded (unstamped) corpus	24.8 s	100,000	untouched
`metadataOnly` + skip on a pre-#1984 corpus	29.0 s	100,000	untouched
`metadataOnly` + skip, all current (no-op)	20.0 s	0	untouched
`metadataOnly` + skip on unstamped, before the fix	24.1 s	0	untouched

:missing=true p50 latency, selective case (subject:missing=true, 0 hits):

State	p50
Upgraded corpus (unstamped arrays)	969 ms
After `metadataOnly` + skip, before the fix	985 ms (unchanged)
After `metadataOnly`	6.6 ms
Pre-#1984 → after backfill	1,222 ms → 1.5 ms

Backfill: about 4–5× faster than a full rebuild, and the index is untouched.
No-op runs: a skip run with nothing to write still costs about 80% of a real run, because every resource is read and extracted. With the fingerprint, the follow-up you listed (pushing the skip filter into the walk) could now use missing_params_fp != current.
Normal rebuilds: no measurable change. The timing spread between runs came from host load, and seeding, which this PR doesn't touch, varied by the same ~25%. The only code change on that path is one Option check per page.

Minor, not changed

Older nodes silently ignore unknown $reindex parameters. During a rolling upgrade, a metadataOnly request that reaches an old node becomes a full rebuild. A sentence in the upgrade steps (send it to an upgraded node) would cover it.

smunini · 2026-10-24T23:53:36Z

Re-test after merging main (`524ac02`)

I merged main (e99b233: #2218, #2231, #2232) into the branch. There were no conflicts, and none of the new commits touch the reindex code. Then I re-ran the tests and the bench with and without this PR, on a MongoDB 7 replica set.

Tests

	main (without)	this PR + main (with)
MongoDB `reindex` / `missing` / `deferred_search_index` integration tests	135 pass	142 pass (the 7 new ones included)
Lib `reindex` tests (`--test-threads=1`)	–	172 pass
REST `reindex` tests	–	20 pass

The skipExistingMetadata fix from cd8a2a8 still holds after the merge. Arrays written by v0.2.5 (#1984), which carry no stamp, are rewritten, not skipped.

Performance: 100k Observations (1.8M `search_index` rows)

Run	this PR	main
`metadataOnly` + `skipExistingMetadata` on an upgraded (unstamped) corpus	21.1 s, `metadataUpdated=100000`	n/a
`metadataOnly` on an upgraded corpus	17.8 s	n/a
`metadataOnly` + skip, everything already current (no-op)	17.5 s, `metadataUpdated=0`	n/a
`metadataOnly` + skip on a pre-#1984 corpus	25.2 s	n/a
Full `$reindex` (main's only remedy)	89–99 s	105–130 s

subject:missing=true p50 (0 hits):

State	Before	After `metadataOnly`
Upgraded corpus (unstamped arrays)	637–918 ms	0.9–1.6 ms
Pre-#1984 corpus (no metadata)	910 ms	1.0 ms

Backfill: metadataOnly is about 5× faster than a full rebuild, and the search index is left untouched (row count unchanged).
Full rebuilds: no regression. This PR's numbers were slightly below main's, which is host noise; seeding alone varied between 92 s and 128 s.
No-op skip runs: still cost about 80% of a real run, because every resource is read and extracted (see my earlier comment).

The rolling-upgrade note from my earlier comment still applies: an older node ignores metadataOnly and runs a full rebuild.

mauripunzueta marked this pull request as ready for review October 22, 2026 13:01

smunini added 3 commits October 24, 2026 13:39

Merge origin/main into feat/2008-mongo-reindex-missing-params

42225e3

Merge remote-tracking branch 'origin/main' into HEAD

524ac02

smunini merged commit 89d7e07 into main Oct 24, 2026
19 checks passed

smunini deleted the feat/2008-mongo-reindex-missing-params branch October 24, 2026 23:54

Navigation Menu

Repository navigation

feat(reindex): metadata-only mode to backfill MongoDB missing_params (#2008) - #2213

feat(reindex): metadata-only mode to backfill MongoDB missing_params (#2008)#2213
smunini merged 6 commits into
mainfrom
feat/2008-mongo-reindex-missing-params

mauripunzueta commented Oct 21, 2026

Uh oh!

codecov Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

smunini commented Oct 24, 2026

Uh oh!

smunini commented Oct 24, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mauripunzueta commented Oct 21, 2026

What

How

Acceptance criteria

Verified locally

Not done / follow-ups

Uh oh!

codecov Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

smunini commented Oct 24, 2026

Review: tested with and without, plus perf

At the PR head (old base 3281155)

After merging main: two problems, both fixed and pushed

Performance: 100k Observations (1.8M search_index rows), MongoDB 7

Minor, not changed

Uh oh!

smunini commented Oct 24, 2026

Re-test after merging main (524ac02)

Tests

Performance: 100k Observations (1.8M search_index rows)

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Oct 21, 2026 •

edited

Loading

At the PR head (old base `3281155`)

Performance: 100k Observations (1.8M `search_index` rows), MongoDB 7

Re-test after merging main (`524ac02`)

Performance: 100k Observations (1.8M `search_index` rows)